1. Open MATLAB R2020a
   Note: Instructions for installing the application and setting up an account are included in UHI MatLab PDF
2. Open a new script and save the file in an appropriate location using a relevant filename.
   Note: that we shall return to this file in future sections so be sure that the file is saved in a secure and accessible location.
3. Using the comment feature in the MATLAB programming language, write a descriptive introduction to your program that uniquely identifies this program, e.g.:

%{
...
Doug Rattray - doug.rattray@uhi.ac.uk
student ID - Lews Castle College, UHI
Date
ANN – Section 2 – Programming Exercise 1
...
%}

Define the structure of the ANN by initialising variables that contain the number of input, mid and output layer neurons.

Note: the number of weights in each layer is the product of the number of sending neurons with the number of receiving neurons.

% Structure of ANN
nInputs = 2;
nNeurMid = 2;
nNeurOut = 2;
nlayer1w = nInputs*nNeurMid;
nlayer2w = nNeurMid*nNeurOut;

Create an empty array (an array of  zeros) for each neuron in which each row represents a separate neuron and three columns deep, where columns shall be used to contain the inputs to the neuron, outputs from the neuron and error values, e.g.:

Also, create two single column arrays to store the inputs to the ANN and the associated target outputs for our training pair.

% Array to Hold Neuron Variables
% 1:input | 2:output | 3:error
NeurMid = zeros(nNeurMid,3,"double");
NeurOut = zeros(nNeurOut,3,"double");
% Arrays to Hold training pairs
KnownIn = zeros(nInputs,1,"double");
TargetOut = zeros(nNeurOut,1,"double");

Create a single column array initialised with random values (between -1 and 1) for all weights that exist between each layer.

Note that we shall use the RAND function in MATLAB, which we will have to multiply by 2 and subtract 1 from to change its range from (0 < x ≤ 1) to (-1 < x ≤ 1).

% arrays to store weight parameters
% Initial value: Random value: (-1 < value < +1)
layer1w = (rand(nlayer1w,1) * 2) - 1;
layer2w = (rand(nlayer2w,1) * 2) - 1;

State the number of training cycles (in this exercise: 5) that we want the ANN to process and the training rate (N).

% Training cycles and training rate (N)
nTCycles = 5;
N = 1;

Finally, create an array into which we shall store the error values calculated in each training cycle.

Note that the array should be sized such that the output from O1 & O2 can be stored for each training cycle.

% Array to store output error of each training cycle
ErrLog = zeros (nTCycles, nNeurOut, "double");

KnownIn = [0;1];
TargetOut = [1;0];

Calculate the Error for each Mid Layer Neuron

For each neuron in the Mid Layer, we shall calculate the error using a rolling sum of the product of each output layer error by its associated weight.

${\epsilon }_{N1}=({\epsilon }_{N2}{\omega }_{N1N2}+{\epsilon }_{N3}{\omega }_{N1N3})\times squashBP$

for m = 1:1:nNeurMid
    for o = 1:1:nNeurOut
        squashBP = NeurMid(m,2) * (1-NeurMid(m,2));
        NeurMid(m,3) = NeurMid(m,3) + (NeurOut(o,3) *
               layer2w(((m-1)*nNeurOut+o),1))*(squashBP);
    end
end

Adjust the weights between the Mid Layer and the Output Layer

We shall now adjust the weights on connections between the Mid Layer and the Output Layer using the Back Propagation algorithm.

${\omega }_{AN}^{+}={\omega }_{AN}+\eta ({\epsilon }_N\times O_A)$

for o = 1:1:nNeurOut
    for m = 1:1:nNeurMid
        layer2w(((o-1)*nNeurMid+m),1) = layer2w(((o-1) *
               nNeurMid+m),1) + N*(NeurOut(o,3) *
               NeurMid(m,2));
        end
    end

Adjust the weights between the Input Layer and the Mid Layer

In the same manner, adjust the weights on connections between the Input Layer and the Mid Layer.

for m = 1:1:nNeurMid
    for i = 1:1:nInputs
       layer1w(((m-1)*nInputs+i),1) = layer1w(((m-1) *
              nInputs+i),1) + N * (NeurMid(m,3) *
              KnownIn(i,1));
    end
end

Designing the Training Cycle

The start point for our training cycle should be set just prior to our definition of the Training Pair (Step 2).

The end point for our training cycle should be after the error log is populated, but before the error is plotted.

for Tloop = 1:1:nTCycles

    % ---------------------------------------------------%

    % 2: Define Training Pair

    % ---------------------------------------------------%

 
    % ---------------------------------------------------%

    % 3: Forward Pass

    %- --------------------------------------------------%

 
    % ---------------------------------------------------%

    % 4: Backward Pass

    %- --------------------------------------------------%

    % Populating the Error Log

    %----------------------------------------------------
 
end

Populating an Error Log

We calculated the output error at the end of each Forward Pass (Step 3) and assigned the value to the 3^rd Column in the neuron array ‘NeurOut’. This value will be overwritten on each consecutive training cycle. As such, we should adapt that section of code to also send the code into our error log, as follows:

Note that this section must be encapsulated within the training loop, created previously.

for o = 1:1:nNeurOut
    squash = NeurOut(o,2) * (1-NeurOut(o,2));
    NeurOut(o,3) = squash * (TargetOut(o,1)-NeurOut(o,2));
    ErrLog(Tloop, o) = abs(NeurOut(o,3));
end

In this step, we shall simply plot the Error Log so that we may visually see how well the network has been trained.

hold on;
for o = 1:1:nNeurOut
    plot(ErrLog(:,o));
end
title('Output error exposed to training');
ylabel('Error value');
xlabel('Number of training loops');